Off-Policy Proximal Policy Optimization

Proximal Policy Optimization (PPO) is an important reinforcement learning method, which has achieved great success in sequential decision-making problems. However, PPO faces the issue of sample inefficiency, due to cannot make use off-policy data. In this paper, we propose Off-Policy method (Off-Policy PPO) that improves efficiency by utilizing Specifically, first a clipped surrogate objective function can utilize data and avoid excessively large policy updates. Next, theoretically clarify stability optimization process proposed demonstrating degree update distance consistent with PPO. We then describe implementation details iteratively updates policies optimizing objective. Finally, experimental results on representative continuous control tasks validate our outperforms state-of-the-art methods most tasks.